Expedited Benchmarking of Social Networking Actions with Faster Data Load Techniques

To benchmark and rate a data store, one might be required to repeatedly load the same benchmark database onto the data store. This may constitute a significant portion of the benchmarking process. This paper focuses on the BG benchmark (www.bgbenchmark.org) and presents three techniques to expedite loading of its database. These include generating the disk image of the database once and reusing it many times, restoring the updated data items to their original value in between experiments, and maintaining in-memory state of the database across different experiments to avoid repeated loading of the database all together. BG may use a hybrid of the proposed techniques when evaluating a data store. When evaluating MongoDB with a million member BG database, we show these techniques expedite BG’s rating of MongoDB from 4 months (123 days) of continuous running to less than 11 days for the first rating experiment. Subsequent ratings of MongoDB with different workloads using the same database will be much faster, in the order of hours (less than 1 day).